Papers with crowdsourcing platforms
BotEval: Facilitating Interactive Human Evaluation (2024.acl-demos)
Copied to clipboard
| Challenge: | Using language models to perform complex interactive tasks is becoming more common with the rapid progress in natural language processing (NLP) models. |
| Approach: | They develop an evaluation toolkit that enables human-bot interactions as part of the evaluation process. |
| Outcome: | The evaluation toolkit enables human-bot interactions as part of the evaluation process, rather than making judgements for a static input. |
Integrating INCEpTION into larger annotation processes (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Annotation tools are increasingly only steps in a larger process into which they need to be integrated. |
| Approach: | They propose to adapt INCEpTION, a semantic annotation platform that offers intelligent assistance and knowledge management. |
| Outcome: | The proposed platform offers a range of APIs and can interact with external services such as authorization services, crowdsourcing platforms, terminology services or machine learning services. |
Chat-crowd: A Dialog-based Platform for Visual Layout Composition (N19-4)
Copied to clipboard
| Challenge: | We present Chat-crowd, an interactive environment for visual layout composition via conversational interactions . system can be integrated with crowdsourcing platforms for both synchronous and asynchronous data collection . |
| Approach: | They introduce an interactive environment for visual layout composition via conversational interactions that supports multiple agents with two conversational roles. |
| Outcome: | The proposed system can be integrated with crowdsourcing platforms for both synchronous and asynchronous data collection and has quality controls on the performance of both types of agents. |
CoRefi: A Crowd Sourcing Suite for Coreference Annotation (2020.emnlp-demos)
Copied to clipboard
| Challenge: | Using a web-based coreference annotation suite, we demonstrate that non-expert annotators can be trained to perform and review coreference resolution tasks. |
| Approach: | They propose a web-based coreference annotation suite oriented for crowdsourcing that provides guided onboarding and a novel algorithm for a reviewing phase. |
| Outcome: | The proposed tool provides guided onboarding and a novel algorithm for a review phase. |
CPJD Corpus: Crowdsourced Parallel Speech Corpus of Japanese Dialects (L18-1)
Copied to clipboard
| Challenge: | Various corpora of dialects have been collected using a well-equipped recording environment due to geographical and expense issues. |
| Approach: | They construct a crowdsourced parallel speech corpus of Japanese dialects using crowdsourcing platforms. |
| Outcome: | The proposed corpus includes parallel text and speech data of 21 Japanese dialects. |
Exploring Amharic Sentiment Analysis from Social Media Texts: Building Annotation Tools and Classification Models (2020.coling-main)
Copied to clipboard
| Challenge: | Existing crowdsourcing platforms do not support sentiment analysis for Amharic, and there are no expert researchers in the area. |
| Approach: | They propose to build a social-network-friendly Amharic sentiment analysis tool using the Telegram bot and collect 9.4k tweets where each tweet is annotated by three Telegram users. |
| Outcome: | The proposed system outperforms existing classifiers in Amharic and other low-resource languages due to the widespread use of sarcasm and figurative speech. |
TurkingBench: A Challenge Benchmark for Web Agents (2025.naacl-long)
Copied to clipboard
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi
| Challenge: | TurkingBench is a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. |
| Approach: | They propose to use HTML pages to perform various annotation tasks on crowdsourcing platforms. |
| Outcome: | The proposed model outperforms other models on the TurkingBench benchmark. |
Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Cognitive psychologists have documented that humans use cognitive heuristics to make quick decisions while expending less effort. |
| Approach: | They propose tracking annotator heuristic traces where they measure low-effort annotation strategies that could indicate usage of various cognitive heurs. |
| Outcome: | The proposed tracking annotator heuristic traces shows that annotators are using multiple cognitive heurs based on psychological tests. |
The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Human evaluation is often considered to be the gold standard method of evaluating a Natural Language Generation system, but its quality is often brought into question. |
| Approach: | They argue that the generation of more esoteric forms of language constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance. |
| Outcome: | The proposed system generates coherent and well-formed text of a particular type, usually given an input such as a prompt, outline, topic, or data. |
Language of Bargaining (2023.acl-long)
Copied to clipboard
| Challenge: | a new dataset is being developed to study how language shapes bilateral bargaining . a recent study examined the use of language in negotiation education . |
| Approach: | They propose a dataset to study how language shapes bilateral bargaining . they recruit participants via behavioral labs instead of crowdsourcing platforms . |
| Outcome: | The proposed dataset is based on an exercise in negotiation education . it shows that when subjects can talk, negotiations finish faster and prices drop . |